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ABSTRACT 

The articles in' this booklet address^ the (juestions 
raised by the Maine Learning Consortium in its efforts to look 
closely at, describe, and assess the use of programs such as the 
Exemplary Center for Reading Instruction (ECRl). Based on interviews 
with six teachers, the first article examines student achievement 
gains through the use of ECRI. Student textbook placement as one 
indicator of increased achievement is addressed in the second 
article, which compares reading placement by grade level in the fall 
of 1976 (before thit use of ECRI ) and in the fall of 1980 (after the 
use 9f ECRI). The thi/d article describes changes in several 
achievement subtest scores for students in one school. The analysis 
of the effect of ECRI instruction on student performance in one Maine 
school district presented in the fourth article includes severail sets 
of information from different sources to be used by the board of 
directors as they 'review programs. The fifth article describes a 
district-wide program evaluation process that evolved from questions 
originally raised from the community. The final article describes the 
Consortium's assessment of changes in teacher's level of use of ECRI 
based on self-reports from more than 350 teachers over 3 years. 
(HOD) 
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PREFACE ^ 

Evaluation has been an integral part of the responsibility of everyone in 
the Maine Mastery Learning Consortium. Teachers have responded to 
survey's and opfened their classrooms for observation. Board members -*pnn- 
cipals, teachers, specialists, superintendents, university- and state depart- 
ment personnel devoted hoi^rs of reading and meeting to the issues,raised 
by evaluation in the Consortium. Staff members have continually responded 
to the results of evaluation activities which indicated needed modifications 
.and next stewln their work. As these studies illustrate, evaluation in the 
Consortium nS4>een an intensive effort of all participants. 
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INTRODUCTION 



Qne of the purpose of the Maine Mastery Learning Consortium was to 
help the sixteen member districts look closely at, describe and assess, tho^se 
of programs such as^Exemplary Center for Reading Instriiction (EC^RI). 
Looking closely'raisp questions. The complexity^ and breadth of ECRL the 
one validated mastefy learning* program used in all Consortium niMiber 
districts, meant that the questions were many and various. To help par- 
ticipants clarify these questions was the first step in the Consortium's evalua 
tion assistance. The^Consortium*s evaluation specialist, a part-time staff 
member resjionsible for both the evaluation of the Consortium and the 
developmertt of evaluation skills for participants, worked with Consortium 
and district staff in collecting, analyzing, and reporting information related 
to these questions. 

The articles in this booklet represent the range of these evaluation efforts. 
Some studies^are quite focused. Catherfne Harding interviews six teachers in 
depth and Vance Keene analyzes student text book placement as one in- 
dicator of increased achievement. Wilder^Unt describes changes in several 
achievement sub-test scores for students 'at one school. The broad report of 
Frances Ambrose to SAD #75*s Board of Directors includes §everal sets of in- 
formation from different sources to be used by the Board as they review pro- 
grams. James Kelley*s article describes a district wide {Program evaluation 
proc^ which evolved from questions originally raised from the community . 
The final article by Sara Mas^ and Jeanie Crosby describes the C(>nsor* 
tium*s assessment of changes in teacJiers* level of use of ECRI based on self 
reports from more than 350 teachers over three years. 

V 

We hope these articles provide ideas for other educators needing^o collect 
information about program impact and worth. 

Jeanie Crosby ^ 
Evaluation Specialist 
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Student Achievemeiit Gains 
^ Through Use Of ECRI: 
Teachers' Perispectives 

Catherine Hardi 

INTRODUCTION 



Most studies related to ECRI describe stident achievement in terms of 
standardized test scores. In an attempt to broaden the description of student 
flthievement and to understand more about/he dynamics of EQRI and stu« 
dent achievement, the Consortium undeitook a small stud> of teachers* 
perceptions of the relationship between ECRI and student achievement in 
their classrooms. The participants were six teachers who had used ECRI for 
at least four years and had a high level of ECRI use as identified by v alidated 
self reports and b> staff observations. Each of th^ese teacher^had taught for 
at least seven >ears and was able to reflect On experiences with student 
achievement before using ECRI as well as while using ECRL 

Each teacher participated in a strudtured interview during February and 
March, 1981. Some interviews were cl)nducted in person at the school and 
others were conducted over the telephone. The interviewer, the author of 
this article, was constant. 



The development of the interview began with questions that teachers, ad 
ministrators, 4nd staff were asking. These, initial questions ivere cross- 
checked with ECRI pods or areas of study to make sure that the interview 
>\as incliisive. Questions used in the interview examine achievement in ski^ 
areas, stuaent independence and motivation, student attitudes and values, 
and application of learning from ECRI to other areas. A oop> of the inter 
view questions is found at the ^d of this article. 



C^herine Harding is AssSciate Director of the Maine Facilitator Center. 



This compilation of teachers* responses Is organized b> the questions in the 
Interview, Many responses were similar and have^been synthesized here 
Specific responses included were chosen for their clarity in representing 
either divergent or common points of view. The teachers 5vho contributed to 
this report were gracious with their time and open in their coipments. All 
statements in the next section represent these teacher responses. Conclusions 
of the author are found in the last section of th6 article. 

TEACHER RESPONSES 

L Specific Skill Development 

A. Oral Reading^ 

All teachers responded that students are more fluent in their oral 
reading. They felt that students know more words and can read 
smoothly. One teacher said that while childrer can read more 
rapidly, their phrasing and expression have declined. Some 
teachers thought that some children feel self-conscious about 
oral reading in small groups and that it is easier to use choral 
V'^ading and emphasize phrasing and expression in mjore tradi- 
tional large group instruction. • 

B. Reading tomprehension 

All teachers saw gains in reading comprehension. One teacher 
said small groupings allowed her to see better when children 
were comprehending what they read. Another said children 
show a good understanding of the main idea and that ECRI is as 
strong as any traditional method in bringing out comprehension 
skills. A tl^ird grade teacher singled out the inferential thinking 
taught in ECRI and said stifdents like "digging beyond the 
literal meaning." A second grade teacher qualified her com- 
prehension gains saying children do better with parts of a story 
than with whole stories because details tend to be lost in extend- 
ed passages. She had to work harder on comprehension skills , 
than other areas. * 

^ / 

C. Word Recognition ^ 

Everyone agreed there were enormous gains in word recogni* 
tion. "Kids seem to absorb word attack skills and recognize new 
words spontaneously . . . everything begins to gel . . . reading is 
fun." Teachers said that children associate similar words and 
. parts of words and they begin to understand how they learn hew 
. words. Once the children have achieved sight recognition, the 
repetition in the program h]elps children retain the words 
theyVe learned. 
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D. Vocabulary Development 

Here again, teachers saw gains. One teacher said her students 
"don't come to school with much of a vocabulary, but ECRI 
helps them learn varied and abundant words afld encourages 
them to use these words in sentences/* 

Teachers also reported that the creative writing component of 
ECRI generates an interest in descriptive and figurative 
.language. Children seem to like learning synonyms, figures ^ 
speech and word meanings as used in ECRI. A major indicator 
to teachers of vocabulary development is the use of new 
vocabulary in writing/ 

E. SpeUing ' L 

Most teachers had solid aphievement in spelling in their classes, 
^he children spell very well - every word they read, they 
spbll." Teachers said that evea multisyllable and irregular 
woirds "are spelled in a first grade classroom if the^y are used in 
the children's texts. The children in ECRI classes have word at- 
tack skills for spelling as well as reading. Teachers felt that 
students know how to apply phonics and see relations of parts of 
a new word to other words ihey know. 

There was some variance in reports of spelling achievement in 
free compositions. A second grade teacher ^aid almost no errors 
appear in stories which children write, but a third grade teacher 
said spelling slipsf in children's free writing. Many of her 
children need to work consciously on spelling ahd seem to spell 
better in structured writing exercises than in free writing. 

F. Creative Writing 

All teachers credited ECRI .with helping develop fluid, prpfi- 
cient writers. The children write more often, usually five times a 
week, and write freely in this program. They can expand on a 
subject and use a variety of sentence TForms and openings even in 
first grade. Tebchers said that children aFe not afraid to sgeeJc 
up while a topic is being formulated or to write down their idleas 
when they are writing independently. 

C. Penmanship 

Teachers said children were performing better over all in pen- 
manship than ever before. The first grade teacher said she had 
to stress control and performance in this skill area because her 
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children were in a tremendous hurry to write as well as they 
^ read. However, the other teachers pointed to ECRFs standards 
for excellence in penmanship and said they obtained pretty 
much what they demanded in this area. Teachers cited specific 
improvements in fine motor control, printing, and in cutsive 
styles. Bfdth ECRIs step-by-step teaching process and use of all 
learning modalitfes were seen as contributing to achievement in 
penmanship. • ^ 

H. Reading and V^riting in the Content Areas I 

Geherally, "children carry over wh^Hhey have learned [about 
their own learning process to other content areas." This rwnark 

) summar&es the teachers* views on reading and writing in con- 
tent areas other than language arts. Children apply word attack 

• skills and "practice time" behaviors to other subjects. Com- 
prehension, spelling and writing skills seem to transfer as well. 
One teacher observed her children making up their own direc- 
tlvesjpr studying social studies and science. **They want to use 
what^ey know in all subject areas." ' 



1. Listening 

, I4ws about a9hievement in this skill area stimulated the most 
Swgreement among interviewees. One teacher said listening 
vwmuch better - "there's something to listen to . . . children 
want to respond and they want the po'sitive reinforcementj" 
Another said listening dropped off after children w.ere in the 
program a couple of years. The similarity of directives and the 
repetition in £CRI seemed to discourage special listening skills. 
Another teacher's point of vielt^ was that listening had improved 
In her classroom but that .listening declined Nvithout constant at- 
tention from the teacher. 

In summary, *all language arts skill areas showed improvement 
in the ey^ of experienced ECRI teachers. Certain skills seemecl 
secondary to some teachers and they had to work harder to ob- 
" ^tain gains iii these areas— comprehension in one second grade, 
oral reading in another second grade and spelling in third 
grade—, but one persort's obstacle was another's most reward- 
ing area. No one mentioned weaknesses in the program's direc- 
tives or basic approach to instruction. Rather, they citecl their 
own inadequacies and said they had to learn more a^out a skill 
area, or give it more time, or work more directly with the 
children in order to make similar.progress in all areas. They felt 
that the comprehensiveness of ECRI covering eight language 
arts skills is a challenge for even the ablest teachers. 
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J. When asked if there were one skill area where ECRI was especially 
helpful, three teachers said creative writing, two said comprehen- 
sion and one mentioned word recognition. Areas that were not 
helpful were procedures rather than specific skill areas. For ex^- 
ple, the requirement for practice time in all skill areas was difficult 
for some to follow. One teacher said she never had time for exten- 
sive practice in spelling.. Another said the language arts skill ac- 

^ tivities in the Teacher's Edition of ECRI were hard to fit into the 
program; however, two teachers were not able to cite any skill 
areas where ECRI was not helpful. 

K. Three additional questions were asked to clarify the basis of infor- 
mation on student achievement and to see how relatively slower or 
faster students achieved in this pro-am. 

1. Teachers felt that most slowerstudents benefit from ECRI. 
^ They gain strong foundations for learning; there is less guessing 
about what they know; and skills are introduced through all 
learning modalities so children have many opportunities to 
learn th^ best way they can. Teachers reported that some 
children have gained as much as two years of growth in six 
months of ECRI in^ruction. 

Only one teacher' fiad reservations about using ECRI with 
slower students. She said some students react negatively to the 
pacing and pressure in ECRI. She has a few students in her 
lowest group who are below the ^thers but who cannot be 
regrouped or transferred to a low6t grade because of teacher 
management considerations. ECRI teachers are encouraged to 
maintain only three instructional groups per class and this 
teacher felt she had closer to seven levels in her classroom. 

a. All teachers felt that ECRI worked well wijJi most of their 
children. One teacher said "ECRI is c/bviously a program for the 
gifted - children can work on their own level and progress at 
their own ratt." Some made adjustments for advanced students, 
but did not abandon the program's basic instructional ap- 
' proach. Some adjustments were; 

-giving gifted children more comprehension work, introduc- 
ing irregular word constructions and fading directives 

-staying ahead of the most advanced students an3 providing 
lots of enrichment activities for them 
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-allowing giftfejJ children to do reading in content area texts 
written at a higher grade level 

-modifyin^the <pape of . instruction and introducing more 
work. ^ ' \ 

> 

IL Student Independence and Motivation to Learn 

Teachers reported that there were several aspects of the ECRI pro- 
cess which contribute to stml^nt independence and motivation to 
learn: 

-orienting children to the learning process and letting them 
know how to achieve mastery 

-providing practice time arid the opportunity to organize and com- 
plete work as they see fit during that time ^ • 

-pacing the children so they make progress and see milestones in 
their learning ^ 

-placing the children on or near individual learning levels so they 
can make steady progress and pass mastery tests often 

-encouraging cooperative learning in small groups 

- * The results of these procedures which teachers perceived are that 
children devel6p good study habits, they work at their own speed and 
pjrogress independently, and they help ^ach other more in class. ECRI 
seems to help students organize their tima and effort; they are less 
distracted and disruptive in clas5. 

■ / 

The only part of the ECRI process that teachers sUw as restrictive is 
"the "lock-stepping*'. One teacher said that after children have been in 
the program a couple of years, they do not enjoy the orientation or 
"lock*stepping" period. 



III. Student Attitudes and Values 

" When asked about student attitudes toward reading and school in 
general, teachers saw positive changes. They said Children like reading 
and language af^, children enjoy practice time and mastery tests,^and 
children are eager to'come to school Teachers also felt that children 
like the behavioral reinforcements in ECRI. One teacher said the 
behavioral aspects of the program gave her rtiore calm and helped her 
and the class "suffer through" the disruptions of some children. Her 
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, students seemed tq know that constructive behavior will pay off more 
^han disruptive behavior. 

Another perceived outcome of the program wds that children exhibit 
a more positive attitude toward' each othpr. When the teacher models 
positi\e reinforcement, the children imitate that positivism. Teachers 
felt that children genuinely care about each othel^? successes. * 

IV. Learning Related Outc6me^ Application to Other Areas 

Teachers said that students' increase in reading^chievement carried ^ 
over*to other subject areas. A grade 1 teacher said, **the good filings 
associated with reading success create receptivity to learn other 
things." Another teacher cited listening skills improving in other 
areas— the children anticipate directives and know they^will have to 
. ' respond to instruction throughout the day. A third teacher mentioned 
• carry over in the students' concern about^ mastery. She uses mastery 
tests fqf .math facts and core concepts in other content areas. 

All teachers said ECRI affected their teaching of all subjects. They 
used E^fl's management techniques, provided direct instruction, and 
elicited responses as a matter of course. One teacher summarized the 
carry over as follows. "My ability to recogpiz^ children's needs and 
teach to those needs is so much stronger. My behavior toward the 
children affects their Achievement in all areas of the curriculum." 

CONCLUSIONS 

In talking with these teachers using ECRI, it became clear that they 
perceived outstanding results for students and teachers with this pro- 
gram. The factors which were seen as contributing to this success^ 
varied, but everyone had something positive to say about ECRI. When 
asked what contributed most to student achievement gains, teachers 
responded: 

-"The children learn how to learn, and the teacher learns how to 
recognize children's needs on the Spot." 

-"The expectation of mastery contributes to achievement gain^. Also, 
the structure of teaching, the efficiency of introducing new words 
and the review built into the pro-am contribute to children's suc- 
cesses." ' 

-"Students* attitudes are better so achievement is better. Also, tfie be- 
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havior management and modeling of tlje learning process by the 
teacher helps children achieve." ' 

•"The integration of language arts skills contributes most to students* 
achievement. Nothing is isolated." 

♦ . 

While all teachers Interviewed said ECRI woiked for them and their 
students, each had adapted the program somewhat to suit individual 
teaching styles or student learning needs. One teacher said she had 
been leery of the program when it was first introduced but would nbt 
teach any other way now. This study offers many examples of the stu- 
dent achievement possible through ECRI but does not propose that 
achievement occurs independently of caring, conscientious teachers^ 
All these teachers made ECRI work for them rather than resigning 
themselves to work for ECRI. 



STUDENT ACHIEVEMENT GAINS THHOUGH USE OF ECRI: 
TEACHERS^ PERSPECTiyE 

INTRODUCnONi 

The quesuom that folW look at ftudcnl achievement In tenw of ipedfic skill acquisition, student in 
dependgfc^ and moQvation, attitudes and values, and applications of learninf behaviors tc other content 
arco, QuesUons were derived from open-ended interviews and surveys with ojen of ECRI The Intent of this 
intcxvleiv is to examine as many indicaton of ac^cvemcnt as possible from a teacher's perspective 

I. Specific JCnowled^ and Comprabeoslon Outcomes 

A. Have you noticed changes in student achievement in the following skill areas- 

1. Oral Reading / 

2. Reading Coiti prehension 

3. Word Recognition 

4. Vocabulary Development 

5. Spelling Improvement 
ft. Creative Writing 

7. Penmanship • 

8. Reading^riting in Content Areas 

9. Listening Skills 

B. Is there one skill area where ECRI hu been especially helpful or undersUnding? 

C. What ways do you measure achievement in your classroom? 

X D How does ECRI affect slower students* achievement gains? What about gifted or advanced 
students? 

^ E. Do you think students are mastering 95% of the skills taught directly through ECRI? 

II. Student Iildependtooe and Motivation to Leara ^ 

A. How does ECRI affect students' independenbe and motivation to learn? 

III. AttitudM and ValiMt 

A. How do students foal about reading/language arts? 

B. Do they enjoy tha competition of mastery tests? 

ERlCie 



C. Do students like wrorklnf at theli own rate oi do tbe> seem to won> where othec children are in 
their readlnp? 

D. How do studenb react to the behavioz modiAcatloa and self-dijdphne encouraged b> ECRI. Do 
they like being in control, being rewarded? 

♦ 

E. Have you noticed better student participation, fewei abaencet, more overall eojoyment of school 
since using ECRI? 

F. Do your students pzhibit a more positive at^tude toward each other? 
IV. laming Related Outcomes - AppUcatioos to Other Areas 

# 

A. Do you think students' increase in reading achievement has affected thdi achievement In othex 
areas? 

B Have you consciously ^ox unconsciously / changed the w a> you teach <Aiket Slk^x. areas because of 
students' achievement in ECRI? 

C. Do you see any dxanges In achiev^eot in these other areas?* 

D. What do you think there is about ECRI that contxibytes ,moct to achlev^ent gains? 
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Comparison of Readings 
Placement by Grade Level in 
Fall 76 Before Use of ECRI 

and in 

FairSO After Use of ECRL 

Yance Keener 

The Lisbon school system began implementation of master> learning 
through the ECRI model in 1977, After three ij^K^.both teachers and ad- 
mmistrators were sure that students were reaSiiig ^Better. 'In looking for 
specific indicators of this change, we realized that there was a pattenupf 
teachers requesting more texts at the higher levels of the Ginn 360 and 720 
senes. In the attached chart which compares the reading placement of 
students in 1976 before using ECRI and in 1980 after Using 'ECRI, the 
changes are apparent. For example, in 1976 no students were using the Level 
12 or 13 reading texts. In 1980, I6V3 % of the 4lh grade students and 58% of 
5th grade students were using Levels 12 and 13. In 1976, 58% of the second 
graders were using books below grade level. In 1980, 54% were using books 
above their grade level in readability. 

In the chart, the reading levels 3-14 have been determined locally by ap- 
plying the Fry readability test and b> comparing these results with the Foxie 
reference. It is important to note that the assignment of reading placements 
as. recordpd here are confirmed as "appropriate" through the use of informal 
reading inventories, standardized reading achievement tests and the recom 
mendations of classroom teachers. Eight of the ten teachers represented in 
the figures for 1980 have used ECBl for at least two years and several for 
three years. Some students new to the district may be using ECRI for the fyst 
time; others* experience with ECRI ranges from one to three years. 

Tlie tomparison of reading placements illustrates that prior to the use of 
mastery learning students entering our classrooms were placed in reading 

Vance Keene is Principal of the Lisbon Elementary School , 
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books "at" or "below" grade level. Reading instruction provided limited op- 
.portunities for students to be instructed be>Ond one instructional level above 
their grade placement. We expected most students to be on or below grade 
level and that is where we "found" them. Toda>, there are often five levels 
of reading texts in an> one classroom with an average of 90% of the class 
reading texts "at" or "above" grade level. In this comparison, we have not 
tned to separate the influence of students* increased skills from the influence 
of the more flexible organization for reading instruction on the rise of use of 
higher level texts. We do feel, however, that use of ECRI has helped cts in- 
crease students' skills and be more flexible in placing students at the ap- 
propriate instructional level. ^ * 

Through experience with master) learning, a lot of things have changed at 
Lisbon ElerAentar> School. We have come to understand the importance of 
placing students at their correct instructional lev el, and w e hav e acquired an*' 
effective means for accomplisHing this through use of the ECRI model. 

COMPARISON OF READING PLACEMENT 6Y GRADE LEV^EL - FALL 76/80 
' 0 LISBON SCHOOL DEPARTMENT 



Reading 
Levels 


Average 
Read- 
abUity 


Grade 2 

76 


80 




Grades 

76 


80 




Grade 4 

76 


80 




Grades 

76 


80 








/ (%) 


/ 


(%) 


/ 


(%) 


/ 


(%) 


/ (%) 


/ 


(%) 


/ (%) 




(%) 


L3 


(?) 


3 (04) 


























L.4 


(?) 


9 (11) 


























L.5 


(10) 


34 (43) 


6 


(11) 


7 


(08) 


















L6 


(2.0) 


33 (42) 


20 


(38) 


21 


(25) 






2 (02) 






1 (01) 
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(3.3) 




11 


(20) 


25 


(30) 
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(08) 














L8 * 


(3 3) 




13 


(23) 


17 


(20) 


13 


(17) 


25^ (28) 






4 (04) 






L9 


(4.3) 
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(11) 


14 


(17) 


16 


(21) 


24 (27) 


14 


(23) 


15 (16) 


6 


(08) 


L 10 


(4 4) 












23 


(30) 


33 (38) 


24 


(40) 


30 (32) 


14 


(19) 


L 11 


(5 4) 












19 


(25) 


4 (0^ 


12 


(20) 


45 (47) 


11 


(15) 


L,12 


(6 1) 


















10 


(17) 




28 


(38) 


L 13 


(7 2) 
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/ of Students 


79 


5« 




84 




77 




88 


60 




95 


73 




/ of Teachers 


3 


2 




3 




3 




3 


2 




3 


3 
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Three years later, the Lisbon Elementar) School has learned through its 
use of ECRI that primar> students cxjuld progress to upper elementary 
reading levels causing students to confront reading matter be>ond their in 
terest and comprehension levels. Responding to this problem, the Lisbon 
teachers havg devised a plan for the coming year. At the primary grades, 
ECRI will continue and placement of students will be determined b> infor 
mal reading inventories and teacher judgement allowing students advance 
ment to the highest appropriate levels of t^e Ginn 360/720 program. This 
reading program represents rapid advanced vocabulary development with a 
balance of instruction in basic comprehension and study skills. 

Beginning at grade four, however, students will begin reading in a "new** 
or "different** series, at an independent reading level. This placement will 
allow th^vocabulary development to cKffnge from a word recognition em 
phasis to an emphasis on usage in written expression. Mastery of extended 
comprehension, study and creative writing skills can take place more readily 
at this independent reading level and students will not be subjected to con 
tent that is uninteresting or inappropriate. 



16 



3 



Changes in 
Student Achievement 

( Wilder Hunt 

» In the fall of 1980, Nobleboro Central School began the use of two master 
learning pr9grams, ECRI and Precision Teaching (PT). Eight staff members 
participated in ECRI training in the summer of 198Q,and four began Im- 
plementing'ECRI strategies in ,the fall. Six staff memb^ continued training 
In the summer of 1981 and used ECRI that fall. Five teachers participated in 
Precision Teaching training in the spring of 1981 and in the fall began to im- 
plement PT in math in grades 1-8 an^ social studies in grades 6, 7, and 8. 

The staff became involved in Mastery Learning through ECRI after visiting 
ECRI clfusrooms in nearby schools and attending awareness sessions. The 
schoors personnel had already spent much <!lme debating issues of mastery 
learning and had many unanswered questions such as: 

What is mastery? 

When has a student reached mastery of a skill? 
How do you measure mastery? 

The ECRI and PT awareness training sessions seemed to answer many, of 
these questions, giving the staff common' ground to work out consistent 
mastery learning approaches for the students. / 

In an attempt to collect information about the possible effect of teacher 
and student involvement in ECRI, the Consortium and the school have 
looked at changes in students* standardized achievement test reading area 
scores betweep spring 1980 and spring 1982. Each student takes the Iowa 
Test of Basic Skills (ITBS) each year. Included in the study w'as each student 
in grades 1 -6 in the spring of 1980 wjio remained in the school in spring 1981 
and spring 1982 The scores of students who moved in or out of the school 
during that time were not used. 

WWer Hunt is Principal and 8th grade teacher at Nobleboro Central 
School 
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school identified three separate ITBS scores as being particulaily im- 
portant: vocabulary, spelling, and composite riding Each student s score 
to each of these three areas was recorded for 1980 before^use of ECM,-for 
1981, and for 1982. The .score? were then analyzed for any significant 
changes. Results of this analysis follow. 

' ' ' ■ 

MEAN TEST SCORES IN NCES ^ 



Test 








Grade in 1980 








• 






1 


o 


3 


4 


5 


^ 6 


N 


Mean 




* 

Vocabulary 


1980 
1982 
Change 


69.47 
63.00 
-6.47 


65.20 
55.50 
-9.7p 


58.00 
57.05 
-0.95 


48.33 
40.42 
-7.91 


,56.06 
53.28 
-2.78 


56.28 
52.17 
-4.11 


97.00 
97.00 
97.00 


58.88 
54.27 
-4.61 


Not Significant 


Reading 


1980 
1982 
Change 


67.35 
61.00 
-6.35 


60.80 
53.90 
-6.90 


52.77 
58.32 
5.55 


42.08 
39.08 
-3.00 


53.06 
56.17 
3.11 


55.89 
53.61 
-2.28 


97.00 
97.00 
97.00 


55.46 
54.68 
-0.78 


Not Significant 


Spelling 


1980 
1982 
Change 


67.29 
56.94 
-10.35 


65.40 
55.80 
-9.60 


57.73 
56.45 
-1.28 


44.00 
38.50 
-5.50 


57.67 
M.44 
-3.23 


54.67 
54.83 
0.16 


97.00 
97.00 
97.00 

< 


57.92 
53.58 
-4.34 


Not Significant 


N ' 


S 


17.00 


10.00 


22.00 


12.00 


18.00 


18.00 









This chart presents ^e mean normal curve equivalent (NCE «=ores for 
vocabulary, reading, and spelling on the Iowa Tests of Basic Skills in 1980 
and 1982^The mea^ are shown for individual grad^. ""J ^^.h 7.h 
all students who were in grades one through six in 1980 (three thrdugh eight 
in 1982) and who took the tests with their classes both years. 

lUs apparent that the change were sm'all. None were statistically signifi- 
cant by^the t test (p > .05). As would be expected with beginn ng scares 
slightiy above the national mean, the majority of changes were slight, non- 
si^ificant decreases. In no instances dldscor^ drop by more *an one sta- 
nce Although the standardized test scores do not indicate a significan 
"range in achievement level, several teachers feel that student ach^ement 
has increased. In conversations, school staff speak of increased indN 
vidualization, greater motivation for some students, and specific students 
whietading has improved. In light of the difference between pro e^ onal 
staff members' judgment and the test scores Nobleboro Central Schoo 
might want to look at other ways of assessing ach^vement^ ^ UbSef 
referenced tests based on local teaching objectives might be "^^d This^rief 
study, like most evaluation projects, raises as many questions it answers. 
But asking these questions clearly, in this case looking directiy at student out- 
comes, is a school's most important step in finding answers. 
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Analysis of Effect of ECRI 

Instruction on Student 
Perforriiance, Maine School 
Administrative District #75 

" ^ France Ambrose 

NOTE: 

The Board of Directort of SAD f75 has been particularly interested in the impact of school pro- 
grams on student achievement. The process for an initial, Jormative studt^ on ECRI student 
achievement in 1979 was developed by a district-wide Committee working u.ith Ms Francet Am 
brose, language arto umtiJtant Jot the distrLt. Members of the committee designed intervieu 
schedules and quesUonnaires\ uirried out the interviews, and participated in tlfp analysis oj the 
tnjormation. In 19S1, the Board asked Jot an additional Jormative report as ajollow up to the 
1979 report. In this process Ms. Ambrose and a mastery learning spe^list Jrom the district 
developf:d surveys based on the earlier interviews and qtiestionnairet and analyzed injormati^ 
from ad^^jnent tesU. Ms, Ambrose then prepared this broad report, presented here in sUgfitly 
edited^ofm, Jor the use oJ the Board. 

INTRODUCTION 

In June, 1979, a report to the Board of Directors of Maine SAD #75 was of- 
fered by this consultant regarding the effect of the ECRI model on language 
arts achievement in our district schools. That report was generated through 
questionnaires, interviews, and an analysis of achievement test data. Con- 
tinued interest in the development of student language arts skills through 
this model resulted in the planning of the present report. A survey of parent, 
teacher, and student attitude with questionnaires was likewise carried out 
in the spring and elementary grade students were coded on spring achieve 
ment testing, a process which will be explained in part two of this report. 

PARTI 

yhe questionnaires were designed to assess the attitudes of adults of the 
sMool community regarding major assertions of the ECRI model. This adult 



Frances Ambrose is language arts consultant for SAD §75 and a member of 
the Consortium's Board of Directors, 
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population included teachers, aides, secretaries, parents, and ad- 
ministrators. Teachers and students who do not use ECRI were not 
survejed. Students weresurve>^ regarding their attitudes about themselves 
• as leameixgnd about their feelings concerning elements of the program. 
Respondents were asked to circle a number, from one to seven to indicate 
their degi;ee of agreement w ith eacli statement. B> circling the numeral one, 
the respondent indicated strong disagreement with the statement and b> 
circling seven, strong agreement. In averaging responses, choices of 4, 5, 6, 7 
were viewed as indicated agreement with the statement. 



Adult Survey 



% tgreement % agreement 
non-teacher ECRI teacher 



1 ECRI provides a workable vehicle to integrate language 

arts instruction. ^ 81 100 

2 ECRI provides structure to develop on-task behavior. 84 • 100 

3 Direct teaching results in greater student learning. 79 86 

4. ECRI develops independent learnen. SO 86 

5. ECRI provides for individual growth rates 85- 100 

6. Students in ECRI develop pride in thc^r accomplishments. 85 ' 100 

7. It is important to emphasize the positive in trying to change 

behavior ^94 100 

8 ECRI stresses writing skilU. 81 100 

9. The gifted can be challenged within ECRI format. 73 79 

10. S{>ecial educatfoD students benefit from the structure of 

ECRI. ' 76 93 

1 1 . I believe students learn best through a mastery learning ^ 
approTch * ^ 77 100 

12. If I were the parent of an elementary school child, I would 
like her/him to be taught in a class taught by a skilled 

teacher. 79 93 

ECRI teachers only. 

13. Implementing ECRI results in a seme of persona! and 

professional growth. 98 

14. I have modified the ECRI format in my classroom. ^ 80 



Comments written on, the questionnaires by parents, volunteers, and 
teachers ranged from critical to praising. This variety of feelings about ECRI 
has characterized our study during the past three years. Some samples. 




2, 



A voluntocn "M> daughter wts having a difficult time reading. Since she has had ECRI» she 
has improved 1 00 % . " 



A teacher. "I still enjo> the program most enjoyment coming from the feeling of being in 
touch with students' needs. Tm convinced that review and practice to mastery is the best 
technique for learning. Students seem to feel a great sense of achievement when (heyVe 
worked to master a word, skill, etc. 'The biggest problem I have is with the negative (and 
often unfounded) opinions of the publia." 

A parent. "Although I agree that writing skills are stressed, what inrijpjrcsses me the most about 
the prograi^ is the wa> m> child comprehends the written language and expresses herself oral- 
ly and in writing. Some rtay be due to natural ability, but I credi|i the ECRI program with 
her rate of progress. I do not feel that she would have progressed at the same rate in a 
classroom where spelling, language, and reading are.tau^t as isolat^ subjects.*' 

Other parents. "I believe this to be the best form of teaching I have ever come in contact with 
My daughter's spelling amazes me . . . Please understand (fiat we are staunch supporters 
of ECRI iind would welcome the chance to elaborate on any of our responses to particular 
questions." 

A parent volt&teer: "There is too much structure ... has an adverse effect on 'on-task 
behavior' I have no evidence to prove ECRI develops independent learners. First of all, when 
a child is already intrinsically motivated, independence plays its part. Second, when a child 
goes from the structure of ECRI to a less structured class, what happens? I feel ECRI is too 
test oriented and too little Joy of knowing and <ioing oriented! 1 1 have never found one method 
of teaching right for all students in a classroom." 

Parent. **M> son had this program In another school system ^ts^yeaI. When I learned that he 
would again be In the program In this tow n, I was grateful, for I feel this is the most beneficial 
program for instructing all types of students, be they exceptional, average, or a difficult 
learner." 



% agreed 
97 
94 
90 
100 
96 



, 93 
84 
88 
84 
81 
87 
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Student purvey 

1. My compositions are better than they were in September. 

2. I un a better reader now than I was in Septembei^. 

3. I know how to go about learning what I need to know. 

4. My reading^vocabuiary has improved. 

5. I understand what I read. 

6. I like: 

a. read-a-story (enrichment reading) 

b. write- a-story (creative writing) 

c. learning words 

d. learning writing skills 

e. study skills 

f. comprehension skills 
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89 

93 
93 

« ^ . ■ ,86 

k. practice time " ^ 



g. diction 

h. conferencei 

i. $pcUlng\ 

J. group\|^cus}lon 



1. mtstery tests 
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Some students' w^tten comments: * , 

"I've always been a good reader, but now in ECRI I learn about reading and study skills 
morc.**^ ^ 

"I would IJke to be in aft ECRI class next year because I like practice time." 

* « 

**! think it has helped me improve my learning skills." < 
-•I feel IVe Improved' a lot of thin^ especially penmanship." 
"I enjoy talking things over." 

*Tve learned a lot in spelling bScause Tm learning words I have never known," 
SUMMARY OF QUESTIONNAIRES 

Adult surveys and comments again, as in 1979/ point to accomplishment 
of the goals on which the ECRI model is based. Although most of the com- 
ments are positive, the extremes in comments are typical and reflect the con- 
troversy this program seems to generate/ Administrators are perplexed 
regarding the cause of the controversy; what one person may despise, 
another praises. Student responses were cojislstently positive. 

PART II 

Teachers in SAD #75 first volunteered to train in and implement ECRI in 
November of 1977. Others have trained as opportunities were? made 
available during the summer months. More than half the teachers who cur- 
rently are knoWn as ECRI teachers have voluntarily taken follow up train- 
ing. All teachers have received on-site assistance and the district has iden- 
tified two mastery learning specialists who are available to continue staff 
development.* There are presently thirty teachers, approximately half the 
staff, who use ECRI techniques in the elementary grades, and three more 
are planning to implement it following summer training. A number of 
teachers who have trained in ECRI and have not continued to be directly 



•Slncf the vortmg aj thu rtport, four oddttioncl ftuutiry Uaming tp4chtittt have ^un identified and trained 



associated with /he model do continue, to use aspects of it in their programs. 
Therefore, students in SAD #75 schools have a range of experience with 
ECRI m their classrooms. This range is reflected' in the coding system 
describe^ below. 

The following numbers of students have been taught through ECRI 
techniques for at least one year during the elementary grades, as of June, 
1981. 



Kindergarten 
Grade 1 
Grade 2 , 
Grade 3 * 



24/82 , 
160/188 
178/208 
219/243 



Grade 4 
Grade 5 
Grade 6 



187/234 
189/235 
176/230 



The students in the above count have been instructed in ECRI pui^y on 
the basis of ^ whether the teacljcr in question has chosen to use this method of 
instruction. The utilizatidiT^ this model is voluntary and after the first year 
of implementation, the decisioji to continue in the model is made by the 
teacher. Teacher fidelity to the model also varies greatly in our district witlf 
those who are attempting to develop the program at various levels of profi- 
ciency. Teacher experience with the model varies greatly. Therefore, while 
this study attempts to report student performance of ECRI students on the, 
C^ifomia Achievement Test (CAT),' it was unable to identify the variation 
in teacher performance within each group. Further, it was not possible to in- 
aicate how well these students performed a year ago. All we can look at is 
how well our elementary student? scored in reading and language arts this 
year. 

* 

Due to the above limitations, nine codes representing student experience 
with ECRI were used to look at student performance in kindergarten 
through grade 6 on the CAT. - 



Code 1. student who is presently in an ECRI classrcram and was in an ECRI classroom dur- 
ing 1979-80 and 1978-79 schooHyear. (Three consecutive years.) 

Code 2 stud^t who is presently in an ECRI classroom and has been in an ECRI classroom 
durinK any other two years prior. (Three years but not consecutive) 

Code 3. student who is presently in an ECRI classroom and was in an ECRI classroom dur- 
ing 1979-80 school year, only. (Two consecutive years.) 

Code 4 student who is presently in an ECRI classroom and was in an ECRt classroom any 
year previously, except 1979-80. (Tv/o non-consecutive years.) 

Code 5. student is presently in an ECRI classroom but has never been in an E^RI classroom 
any three years previously. 
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Code 6i student is not in in ECRI classroom *but his been In one ECRI classroom any three 
years previously. 

Code 7. student is not in an ECRI classroom, but has been in an ECJRI classroom any two 
years previously. 

Code 8: student is nof in an ECRI classroom but was In an ECRI classroom for one year 

previously. * ' . • - 

\ • .* - 

Code ^: student is not in an ECRI classroom and never has Been. 



RESULTS OF TESTING: 

V ,* 
The following conclusions are based on an analysis of coded group perfor- 
mances using mean scaled scores of these groups and local (total) mean 
scaled scores. Mean scaled scores may be understood to be a translation of 
raw scores (number correct) to an. equal interval scale ranging from 000 to 
999. Mean scaled scores on the CAT change for each subtest as they reflect 
the average performance of the norming population; therefore, our local 
means will change for each subtest, also. Local mean scaled scores are 
averages and therefore include the scores of each of the coded groups 
Students are placed heterogeneously in classroom groups with great care 
given to maintaining a range of ability and achievement in each group. 

Kindergarten: There is only one ECRI kindergarten in this district. Mean scaled scores in the 
pre-reading battery of this group of students who have had ECRI for one year (Code 5) was 
higher than other kindergartens (Code 9) and was 27 points above the local mean 

Grade One: All but one of the first grades are identified as ECRI. This group of 160 first 
graders (Code 5) outperformed the non-ECRI classroom (Code 9)'by fifteen points in reading and 
thirty-three points in language. 

Grade Two: Code 3 second graders, students who have been in ECRI classrooms f^iiwo con- 
secuuve years, received scores in vocabulary and comprehension equal to or better thanthe local 
average but did not perform as well in these subtests as Code 9 students who have never had 
ECRI. However, Code 3 students and Code 5 students (this was their first year) outperformed 
Code 9 in spelling, languag<?mechanics, and total language scores Code 9 second graders we«? 
an intact classroom of 22 students. 



Grade Three: Code 1 third grade scores reflect performances of students who have been in- 
structed in ECRI since entering first grade. Scores in reading and language arts were higher than 
Code 9 group by nineteen scaled score points and twenty-one scaled score points, respectively 
The scores of this group ojf students were also superior to the local mean in all reading Und 
language subtests in a range of ten to twenty points. This profile was consistent for third grade 
students who had been in ECRI two years, whether they were currently in an ECRI classroom or 
not (Co({e 3 and 7) . This grade is perhaps the most pertinent one to study when looking at the im- 
pact of ECRI in our district, as it contains students in all the codes except 6 as well as students 
who have never had ECRI and those who have never had any other type of instruction, 

Grade Four: Students whp had had ECRI one year outperformed otfjer coded grade four 
students. The effect of ECRI training was difficult to assess at this grade except to conclude that 
students who had had one year of ECRI outperformed those who had had none Code 6 students, 
^ *^3wcver, outperformed other groups who had been ijastructed in ECRI two or three years^ 
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Grade Five. Students in Code 1 and 2 outperformed othei coded groups, often by more than 
twcnt> scaled points According to this coded profile, the impact of £CRI in the intermediate 
grades appears tolx cotisidetable Reading comprehension and language expression, majui goals 
of language arts instruction in the intermediate grades, superior in these coded 1 and 2 
groups: 

Grade Six. Coded ^ups for grade six are 5 through 9. Code 5 represents one classroom of su 
graders «»hv had EC HI instruction for the first time this year. This classroom outperformed all 
other coded groups in grade six except in spelling. This classroom also received scaled scores in ex- 
cess of the local mean in reading and language. Coded group 6, students who wcfe/iot currently 
in an ECRI classroom but had had training for three years previously , also were superior tu Code 
9 in comprehension, language mechanics and expressions as well as total language scores. Code 6 
students were superior in spelling. 

SUMMARY 

&n the spring, 1981, California Achievemenl Test, SAD #75 students who 
have been instructed in ECRI generall> /Outperformed students who have not 
participated in this approach. Initial . pre-reading skills taught in 
kindergarten through this approach appear demonstrabl> stronger than in 
non E!CRI classes. Priinar) reading and language arts instxJUction pursued 
through this rfiodel, as assessed on the CAT, appears to grow stronger 
according to the nlimber of >ears students have participated in the model. 

.Iixdeed, analysis, of grade three results points to the significant impact of 

, ECRI on Achievement. 

Concern about ECRIs stress on language mechanics and reading com- 
prehension seems unfounded as students in these coded groups in primar> as 
well as intermediate grades excelled in mechanics and comprehension. 
While there is no clear pattern, spelling instruction in some interm^iate 
grade ECKI classrooms may need further development. 

Grade fouyresults do not follow the,patter;i of the other six grades. This 
writer has no/reason to offer at this tinie.liowever, the pattern of grade five 
and six ECRI groups suggests a strong correlation betw een thjs approach and 
"superior reading and language arts achievement scores on the intermediate 
levels of the CAT. 

Readers of this report should be sure to note that all ECRI goals are not 
and cannot be measured on a standardized test. Developers of this^rogram 
as w ell as the teachers trained in it stress learner attitude and dev elopment of 
stud> skills as a major goal. Teacher^, particularly , note how this approach 
. teaches students how to learn and ho^w.fb Icftpw when the>\ e learned. As on- 
ly one master) learning system, ECRI i^cvejrtheless stresses high expectancy 
student and teacher performarice as long as one is given quality instruc- 
tion and sufficient practices. H^nce, ifeacners in this model point to the 
development of learner attitude as one of the ECRI strengths. 



The development of creadve^yriting sjdlls stressed in ECRI classrooms 
isn t assessed on the CAT. Parents and teachers see this aspect of ECRI as 
particularly important. 



At this point in the implementation of ECRI in SAD #75, the model has 
begun to be adapted to local needs. Teacher personality has shaped its adap- 
tation as have the needs of individual students. While there are critical 
elements which must be maintained in a master)' learning classroom, ECRI 
classrooms district-wide possess their own individual flavor. This mastery 
learning model also has provided the district with a springboard for staff 
development in its teacher-led seminars and demonstration classrooms Fur 
thermore, implementation of this model has generated interest in master>' 
learning as' a concept, as well as interest in other mastery learning models 
Therefore, the impact of ECRI on this district cannot be measured by stan- 
dardized test results alone. Man> of its positive effects can only be seen in the 
spin-offs in teacher, administrator, and student attitudes about their own 
ability and the ability of others to accomplish personal goals. 
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Working Toward Agreement: 
The School and The Community 

James Kelley 

From the time teachers In SAD #59 began to use ECRI, parents* Ideas and 
feelings have been a part of the program assessment process. Families and 
the larger communit) continue to be interested in reading and vvriting. They 
have an important stake in young people learning to use the language well. 
The> hear children's comments about **what happened at school today.** 
Families know when children can read the newspaper and watch children*s 
approach to homework, etc. Parents also have a sense of and questions about 
whether an instructional process— what happens in school — is congruent 
with their values related to learning. 

y ^ 

For several ye^rs parents* questions, ideas and feelings about ECRI use 
were collected through informal conversations Svith individual teachers, ad- 
ministrators, and school board members. As representatives of the communi- 
ty, school board members received information and made decisions about 
ECRI, including the allocation of funds for continued training, the purchase 
of materials, and other support. In the fall of 1979 after 2 years of ECRI use, 
the school district used a formal survey process to gather information and 
responses from all parents as part of the ongoing program assessment pro- 
« cess. 

The survey questionnaire was developed for the specific purpose of prA- 
viding a way for all parents to share their thoughts about the ECRI pro- 
gram, their feelings about their child's reading progress, and their percep- 
tions of their child*s feelings about the program. A copy of the questionnaire, 
which was originally used in Portland and revised for use in SAD jf^59, ap- 
pears at the end of this article. 



Over 70% of the parents responded, indicating an active interest in their 
child*s learning. This high response r^te on a questionnaire sent home with 
arid returned by students is an illustration of the seriousness with which 
parents in SAD j|f59 take their dialogue w ith the school and of their trust that 

/ 
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their responses are valued and listened to. Analy-sls of the responses yielded 
*the following conclusions: 

^alChou^ the siAvey wis completed in early November, more than two- thirds of the 
parents responding had already talked with their child's teacher about ECRI 

—most parents felt that their child enjoyed 1£QB1 and using language arts skills 
•89% of parents responded that their child enjoyed reading instruction and working on 
• mastery tests 

•85^ of parents responded that their child enjoyed reading at home and creative writing 

— Ojost parents felt that their child was both making good progress and was aware of that 
progress 

•91 % of parents responded that their child both w as making good progress in reading and 
seemed to be aware of their progress 

•93% of parents responded that their child in^proved in spelling ability 
•91% responded that their child improved ffl handwriting 

About one-third of the respondents offered comments on the program in 
addition to responding to the defined questions. The extensiveness bf the 
comments indicated that' parents were closely observing the program. 
PosiUve comments ranged across the program and often expressed apprecia 
tion of the program and support for the schools. Negative <X)mments focused 
on mastery tests and speed of response desired in master)- tests. The district's 
response to parents* ideas and concerns was broad. The monthly newsletter 
to parents, "Elementary Echoes," carried articles about ECRI and other 
programs. Half-day, grade level teacher meetipgs on ECRI were scheduled 
to share ideas and materials and to discuss issues raised by parents about 
ECRI strategies such as mastery tests. For example, the inclusion of difficult 
or foreign words used in the reading on mastery tests was moatfied through 
these meetings. 4 ^ 

Two years later in the fall of 1981 as the questionnaire was being revised 
for a second survey of parents, a group of parents came to the school ad- 
ministrators with concerns about ECRI. In general, these concerns extended 
the negative comments on the initial survey and included these parents* 
perceptions about: 

—the use of timing in the program, especially with regard to mastety tests 
—the drill emphasized in the program which seemed to "turn ofT some students 
—the desirability and the inconsistency of the emphasis on "master/* 
—the heavy emphasis on spelling and the neglect of the spelling of common words 
—the emphasis on rewiards and the resulting competition among students 
—the role of oral reading and response a^ the> affected other students involved In the learn 
ing^process 

—the advisability of using one approach to the language arts instructional program 
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These parents articulated their concerns in a letter containing more detail* 
ed and organized Information than would have been obtained through the 
survey. At this time, the assessment process moved from informal conversa- 
tions and pencil and paper surveys to a personal process conducive 
parents', teachers', and administrators' mutual e^iploration of ECRI and its 
use in SAD #59. During the next four months, the groups involved were part 
of an assessment process w ith two concurrent components of parent meetmgs 
and teacher meetings. 

All parents were invited to a series of meetings about ECRI. Approximate- 
ly twenty five parents attended the initial meeting where they stalked infor- 
mally, developing a list of their concerns with the program. Interest in con- 
tinuing the conversation was high and a sy-^tem was established for inform- 
ing people of the next meeting. The second meeting held two weeks later 
provided information on ECRI and its teaching strategies followed by a 
question and answer period with a training specialist from the Ma^ne 
Mastery Learning Consortium. At the end of this second group meetfhg, it 
w as clear that a number of parents had specific concerns and were frustrated 
in group meetings by the lack of time to deal with individual problems. 
Therefore, the elementary supervisor set up individual conferences with 
each of the 30 parents who wanted to talk privately about their specific con- 
. cems. After the individual conferences, parents met for the third time as a 
group. The outcome of this last in the series of group rAeetings was that 
parents wanted flexibility in the program and wanted to know the extent to 
which teachers could and would modify the program for individual leari^f^g 
and teaching styles. 

Meanwhile, administrators and teachers were meeting together in grade- 
level groups to discuss and assess their satisfaction and concerns with this 
process of teaching language arts. An important topic was the extent to 
which individual teachers felt they had permission to modify the program. 

This mutual concern about program modification was the focus of the 
next segment of the assessment process. A task force of teachers from each 
grade level met each Monday after school for five weeks to address issues 
raised by parents and teachers. Program modifications which responded to 
parent and teacher concerns were developed and included in a draft report 
for teachers. In a workshop, SAD fSQ teachers in grades K 7 reviewed and 
approved this report. They were particularly pleased to see possible options 
or alternative teaching strategies described in writing. 

-? ^ 

A copy of this report was then mailed to each parent who had signed the 
letter of concern four months earlier. After reviewing the report, the leader 
of the concerned parents' group sent ^ letter of appreciation for the respon- 
siveness of the school administrators. 



This collaborative assessment process regarding use of ECRI has had 
results valued b> both parents and teachers beyond the final report. A broad 
review of language arts scope and sequence and selection of a new basal 
reading series is planned for the next school year. A process for successful col 
laborative assessment involving parents, teachers and administrators in SAD 
jf59 has been established. Time to discuss problems and develop solutions, 
energy to work with both groups and individuals in different formats and 
the willingness to listen to different points of view have all been acknow^ledg 
ed as important in this process. The next stop in the continuing assessment of 
SAD ij^59 programs is sure to build on these learnings from ECRI. 



Your point of view ts a parent Is important. Please check the appropriate column. 

Yes No 

Have you talked to your child's teacher about ECRI? • D D 

Has your child: 

1. Enjoyed reading instruction? 

2. Made good progress in reading? 

3. Improved in spelling ability? 
j^A. Improved in handwriting? ' ^ 
The passing of Mastery Tests is an important part of the program. 
Does your child: ' 

1 Discuss the tests at home? 

2 Seem to be aware of his/her progress? 
3. Enjoy workinji%n mastery tests? ^ ^ 

Does your child • 

1. Enjoy creative writing? 

2. Enjoy reading at home? 
Please add any other comments you may wish to share with us. 
Comments. 



□ □ 

□ □ 



□ □ 

□ □ 



□ □ 

□ □ 
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ECRI PARENT EVALUATION QUESTIONNAIRES 
Summary of Recpocuef * 

West. Ave. Old Pt. Ave. Seb. Rule Atb. Elem. Som. Acad. Junior High 

Yes . No Yes No Yes No Yes No Yes No Yes No 

Have you talked to 
your dilld's teacher 

about ECRI? 17 5 93 42 57 31 21 7 12 8 7 75 

Has your child: 

1. Enjoyed reading i 
instruction? 22 - 123 12 75 13 31 - 17 5 70 12 

2. Made good progress 

in reading? 21 - 125 6 79 7 29 1 14 5 72 7 

3. Improved in spell* 

ing ability? 24 1 111 7 82 5 29 3 15 4 ^ 77 6 

4. Improved in 

handwriting 21 - 124 8 76 13 27 1 17 2 68 10 

Does your child: 

1. Discuss mastery 

tests at home? 20 1 126 11 73 25 27 4 14 6 50 36 

2. Seem to be aware ^ * 
of his/her 

progress? 22 - 109 8 79 9 28 3 17 3 72 9 

3. Enjoy working on 

mastery tests? 19 - 122 9 73 15 29 1 17 4 69 13 

Does your child: 

1. ^njoy creative 

writingV 18 - 122 12 67 19 30 2 15 4 66 18 

2. Enjoy reading 

at home? 23 - 119 11 71 17 30 1 16 3 57 25 

*Out of 553 questiomuUrea distributed, 391 were returned: or 70.70% 
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Changing Teacher Behavior 
Through Training and 
Classroom Follow-up 
Assistance 

> Sara Massey and Jeanie Crosby 

i 

INTRODUCTION 

The Maine Master) Learning Consortium of sixteen school districts pro- 
vided training and classroom follow -up to teachers and administrators in the 
implementation of the master> learning instructional models of Exemplar) 
Center for Reading Insthiction (ECRI), Precision Teaching (PT), and Con- 
ceptually Oriented Mathematics Program (COMP). The goal of Consortium 
services was to increase teachers* level of use of the critical teaching 
behaviors related to the five elements of the Consortium's definition of 
mastery learning present in these validated instructional programs Higher 
levels of use means using more of the critical behaviors more often. A large 
part of Consortium resources Nvere allocated to support districts in their use 
of ECRI, the only program used in each member district. Therefore, the 
focus of this evaluation study was the level of use of these critical teaching 
behaviors in ECRI. 



The need for a consortium evolved during the years after the Maine 
FacUitator Center (MFC) introduced ECRI to Maine districts in 1977 78. 
Two school personnel attended an ECRI awareness session anc^encouraged 
the MFC to ^obtain training services for interested teachers and ad* 
ministrators. An ECRI consultant in New Hampshire provided early train- 
ing in ECRI to school personn^ In Maine in 1978-79. As the number of 
Maine districts adopting ECRI grew, the need for an in state ECRI trainer 
became apparent. A Title IV-C grant to Union #30 (Lisbon) supporting 
ECRI Implemei^tation allowed the hiring of an ECRI trainer for work in 
Maine during 1979-80. Services were delivered through a combination of 
■ * 
Sara Massey is Manager oj\ the Consortium. Jeanie Crosby is the 
^^onsfirtium's Evaluation Specialist 
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local funding from twenty two districts and Title IV C monies. Two of these 
districts had also adopted PT. The need for training and foWov* up services 
related to mastery learning programs w as increasing and beginning to ex 
pand beyond the initial focus on ECRI. 

Representatives of sixteen districts met in the fall of 1979 ,to discuss the 
development of a consortium, a formal association of schools w ith a common 
interest and need for the purpose of sharing resources to meet that need. In 
\Sie spnng of 1980, these districts agreed tp create a consortium and the staff 
of the MFC with assistance from the ECRI trainer suj^mitted a proposal to 
the federal Basic Skills Improvement Program for partial funding of a con 
sortium. This proposal was funded for two jears, creating the Maine 
Mastery Learning Consortium. 

The fheoretical foundafion for the work of the Maine Master) Learning 
Consortium was based in the research on mastery learning. The 
Consortium's evaluation specialist in a review of the research on master) 
learning identified the following critical teacher behaviors: 

1. Teachers plan curriculum and instruction to meet specific objectives. 

2. Teacher^ spend significant time on learning tasks related to the objec 
tives. 

3. Teachers use effective instructional practices. 

4. Teachers continually assess student progress. 

5. Teachers implement corrective procedures quickly. 

The services provided by the Consortium to assist ECRI teachers in ac 
quiring the identified teacher behaviors included: 

1. school-based assistance 

a. demonstrating identified skills teachers want to see modeled 
*'b. team-teaching with a teacher ib improve usage of a technique 

c. observing a teacher to diagnose and prescribe more effective ways to 
implement * 

d. conferencing with a teacher to answer questions and provide feed- 
back ^ * 

e. coordinating a visit by a teacher to another teacher s classroom 

f. conducting 1-2 hour miAi* workshop's on soine skill reques,£ed by 
several teachers**' 

g. doing jam sessions to present information, answer questions and 
concerns, share information, and make materials 

h. doing awareness sessions for parents, administrators, aides, or other 
teachers to help them uncjerstand the program 

i. assisting in designingJocal evaluation studies 
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2. regional as^istanbe 



a. conducting awareness presentations on programs not yet adopted bj 
a district 

b. training sessions 

3. statewide assistaifce 

a. coordinating conferences on specific problem areas 

b. implementing workshops for teachers at various grade levels 

c. training local mastery learning specialists to continue services at 
district level ^ 

d. disseminating a teacher-oriented newsletter 

The Consortium provided these services through a staff of five, a project 
manager (40%), a training coordinator, a trainer, an evaluation specialist 
(20 % ) , and a secretar> . The Consortium operated through a Board of Direc- 
tors representing each member district, the University, the State Depart- 
ment of Educational and Cultural Services, and the MFC. The ^oard pro- 
vided information and made policy decisions to ensure successful deliver) of 
qualfty services to help teachers increase their level of use of the critical 
teaching behaviors as present in the validated master) learning programs. 

EVALUATION METHODOLOGY 

The Consortium's evaluation design was based on two criteria: 

'—the evaluation s indicators of success would be based on changes in 
teachers* level of use of critical teaching behaviors related to the ele- 
ments of mastery learning 

—the evaluation s activities would enhance rathe^ than detract from 
the dally work of the Consortium staff and its participants 

The final design included a series of three teacher self-reports which 
would take 10 minutes or less for response and set of two classroom observa ^ 
tions of a small sample ^^hich would collect information for the validation of 
the splf-reports. 

The ^^If-reflorting instruments were developed using program descrip- 
tions, training materials, conversations with program developers, and 
review by experts in mastery learning. Members of the Consortium's Board 
of Directors representing each member district distributed and collected the 
self-reports from teachers using ECRI. Self-reports were dbtained from 
teachers using ECRI in September '80, the second month of formal Consor 
tium operation. Observations and self reports of ECRI users were obtained 



in March *81 and March *82. The three sets of self-reports completed by the 
participating teachers and the two sets of classroom obser\ations conducted 
b> trained observers were designed to assess the variables of teachers 
behaviors in sufficiently common form to permit comparison of the self- 
ratings with those observers. These behaviors 'include frequency of use of 
strategies such as specified time ratios, mastery tests, informal reading in- 
ventories, group discussions, individual conferences, groups of directives, 
response to positive behaviors and response to negative behaviors, extent of 
record keeping, and procedures for grouping. A cop> of the self-report in- 
strument appears at the end of this article. Level of use definitions from A 
(high) to D (low) were developed using these behaviors found on both the 
self-report and observation instruments. 

The self report instrument foriped the basis of the observation instrument 
which was also reviewed b> experts in mastery learning. The observers came 
from a group suggested b> Board members as persons with training in ECRI, 
interest in mastery learning, and a desire to refine their skills in observation. 
These eleven reading consultants, principals, and teachers met 1) to train in 
the use of the instrument and 2) to assess ii^ter-observer reliability in use of 
the instrument. A majority of this group of observers participated in the sec- 
ond set of observations in 1982. Inter -observer a^eement was obtained to a 
significant level on all sections of the instrument. 

Another major element of the training was to increase understanding of 
the entire study for observers. It was expected that observers would give 
clear, consistent responses to the teachers being observed about their role in 
the study , how they were selected, the usefulness of the information from th^ 
observations, etc. The topics of making the initial contact, entering the 
classropm, ^nd giving feedback were addressed through role plays until the 
observers were comfortable and effective in these tasks. 

An initial sa^nple of twenty teachers in grades 1-6 from eleven school 
districts to be observed was chosen randomly from each level of use in pro- 
portion to the percentage of the total respondents in each level in fall '80. 
One percent of the self report respondents were in level A (the highest), 13 
percent in level B, 19 percent in level C, and 67 percent in level D (the 
lowest). This observation sample remained constant even though the propor 
tions of teachers at each level changed in each set of self-reports. 

The evaluation specialist contacted each teacher in the random sample to 
describe the study and to obtain his or her permission to be observed. Of the 
26 teachers contacted, only four (three from one school) did not wish to»par- 
ticipate. Letters following up on the phone calls were sent to the 22 willing 
participants and their Board representatives. Because of illness and schedul- 
ing problems, two teachers in the sample were not observed in *81. In '82 one 
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of the original group had moved and was not observed again, while one of 
the original group who had not been observed in '81 because of illness was 
observed. 

V 

Analysis of the observations and self- reports of the sample of teachers in- 
dicates that the self-reports are v alid representations of what teachers are do- 
ing in the classroom. The two components, behavior management and 
number of directives used, which did not show statistical significance in the 
comparison of spring *81 self-reports and observations were refined and tried 
again in *82. This time a statistically significant correlation was found be- 
tween teachers* reported and observed use of groups of directives. The only 
area of the self-report which has not been validated is response to positive 
and negative behavior. In fact, many teachers rate themselves lower on this 
strategy than observers. This concern generates the most comments on self - 
reports and is clearly an area which teachers are continuing to develop. 
These two sets of observations had important outcomes in addition to 
collecting information. Observers gained refined skills, had a chance to visit 
classrooms outside their districts, and the opportunity^ to talk with other 
ECRI users. Teachers observed reported a sense of important participation 
in an evaluation study and pleasure in having a possible source of informa- 
tion about both the study and ECRI in their classroom. Observers and 
obse/^ees both reported initial nervousness and growing comfort. The rich 
contextual comments written on both the observations and the self reports 
are reported in the following section as "findings*' along with the more quan 
tifiable checks and ratings. 



FINDINGS 

More than 350 ECRI users have provided self reports from which these 
findings are drawn. 272 in fall '80, 236 in spring '81, and 199 in spring '82. 
The spring '81, self-report added a number of assistants, aides, junior high 
teachers, and resource room teachers who use ECRI in different time 
schedules and various grouping patterns. Of the spring '82 respondents, 110 
had also responded to the initial self- report in the fall of 1980, It is estimated 
that almost all ECRI users in member districts responded to these self- 
reports, thus providing a comprehensive picture of program use within the 
Consortium. 




Strategies 

Mastery tests and IRIs are used almost universally. However, the use of 
small group discussions and individual conferences varies greatly . In order to 
understand better the relationship between the reported and observed low 
use of these str ategtes by some teachers and the vary ing time schedules used 
by some teachers, a question was added to the '82 self report. Teachers were 
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aslced if the> used the 1.2.1 ratii^if'afiy^gfjy^eekl) and to describe their actual 
time schedule if it varied from the prescribed ratio. It is clear that teachers 
are exercising judgment in working abound time constraints. Group discus- 
sions seem to have the lowest priority. Some teachers report leaving out 
back up skills. Others have problems organizing practice time within their 
schedules. Reorganizing is demanded at the upper grades where the> are 
departmentalized. The differences in time allocated to the program either 
by the individual teacher or the school's schedule are wide and must be 
described and accounted for in assessment of program use. For example, of 
the spring '82 respondents, 29% of those at level B used ECRI 3-4 hours a 
day while 29% of those at level D used ECRI for 1 hour or less and 58% of 
those at level D us^ it less than 2 hours. 

Directives 

There are teachers who report using all the directives, there are teachers 
who report using none. The directives most used are those for teaching/iew 
words. In the 1981 self-reports and observations, the actual number of direc- 
tives used w ere in close agreement. In the 1982 self-reports and observ ations, 
use of the eight groups of directives were tabulated rather than the actual 
number of directives used. There was significant level of agreement between 
the number of groups of directives used in the self-reports and the observa- • 
tions. - , ^ 



A number of teachers commented on the self- reports that they do not use 
the directives "exactly". Several others stated that either they or their 
students, especially in upper grades, found the directives "boring" after a 
time. And one stated that she used the directives only for social studies as she 
was riot using ECRI that yearl 

Management 

Again in the third set of responses, man> teachers chose to comment on 
their concerns about reinforcing positive behavior and ignoring negative 
behavior. These responses differ from the '80 and '81 responses in that there 
is less questioning or agonizing about these techniques. Rather, teachers^ 
stated that they tr> to reinforce positive behavior, that they are doing it 
more, and that the> have no concerns about it. Comments about ignoring 
negativ e behavior are more calm and less strident than in *80 or *81. Instead 
of saying wh> the technique will not work, teachers described the times at 
which they do respond to negative behavior. In general, these comments are 
less defensive and reflect teachers' experiences with use of the techniques. A 
number of people also commented on their use of assertive discipline tech 
niques in relation to ECRI. 
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Satisfaction 

Many respondents used this opportunity in the *82 self reports to praise the 
program - "Hooray , it worksl" Others explained the complexities of their tise 
of the program and their satisfaction with it. There are comments like **it*s 
great for creative writing," "it fosters independpice," "students who aren't 
independent have trouble," etc. Some re$pondents state that they are 
satisfied with the program now because thej have^modified it to suit the(r 
needs. In other words, they are not satisfied with wie wjifcle program, but 
they are satisfied with their own version. ^ 

There were also several explanations about the level of satisfaction with 
the program assistance received from others. Some people differentiated be- 
tween assistance from ECRI teachers and from non-ECRI teachers. A 
number of teachers commented that th^y had received help when they asked 
for it, but that they rarely asked. For the first time, respondents mentioned 
the local specialists, saying they were helpful. 



Help teaching comprehension and study skills was the greatest area of 
need as in '81. Managing time in making materials, organizing practice 
time, and finding time for small group discussions was another primary area 
of need. There is also a strong relationship between relati>^ly new users and 
statement of specific needs for assistance. Users with longer experience tend 
not to state needs for assistance with specific groups of directives, for exam^ 



Most important perhaps were many comments that described general 
satisfaction and comfort with use of the program. These respondents men 
tioned that they hoped grade level and other cross- district meetings or brief, 
workshops would continue. It appears that many of the spring 'Sr 
respondents who >v anted to reflect on, assess, and make decisions about,their 
use of the program have done just that. In 1982, there seems to be a large 
group of "mature" users who have made ECRI theirs. 

Levels of Use 

The assessment of change in level of use of the elements of mastery learn 
Ing is a primary purpose of the Consortium's evaluation. A "level of use" 
definition and formula was developed which include frequency of use of 
basic ECRI strategies such as mastery tests, time ratios, discussion groups, 
frequency of use of groups of directives, and extent of record keeping. Each 
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of these areas qf the self«reports was validated observations. The^lf; 
reports yielded the following levels of use: • ^ / 

FaU '80 ... ^ ' 



Level A . , Ipewon (-1%)' 

Level B ' 37 people (M%)t 

Level C 52 people (19%) 

Level D ' 180 people (66%) 

Spring *81 ' , ( . 

Level A 5 people-' * (2%) 

• Level B . * fitf people (11%) 

Level C ^ 76 people (32%) 

Level D ^129 people (55%) 

Spring*82 . , , - 

Level A 7 people ( 4^#) 

Level B) 44 people ^(22%) 

Level or * 114 popple (57%) 

^ Level D 34 people (17%) 



As the percentages indicate, there has been a statistically significant 
change in teachers^ level of use of ECRI during the Consortium's operation. 
Proportions of teachers reporting at levels, A, B, and C increased whijie the 
proportion at level D dropped dramaflcalTy. At the same time, statistical 
analyses of the changes in level of use of the 110 individuals who responded 
to self-reports both in fall *80 and spring '82 does not indicate a sipificant 
association between individual change aid the years of the Coi^ortium's 
operation or the amount of training. There is, however, a significaj^ associa- 
tion between increases in level of use and year^ of use of the pro-am. 

CONCLUSIONS 

-\ 

It appears that during the Consortium's operation the large portion of 
member districts' staffs which were using ECRI in bits and pieces began to 
use more of ECRI more often. Many teachers came to use ECRI more exten 
sively^but also integrate^ an4 modified tlie progr^ within their classroom. 
Additionally, the comments of users indicate that they have a greater sense 
of control over 'the program and that they have made the program theirs. 

I 

Central to the Consortium's evaluation process was die belief tjjfllit evalua 
tion is an appropriate topic;, for all participants and that every one can con 
tribute to evaluation. The SohOTrtium devoted a large amount of resources 
in terms of energy, time and exj^rtlse to evaluation. The staff spent many 
hours helping develop instruments and reporting to participants. Board 
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members devoted several meetings to the topic, and ECRI users responded 
several timfe to^ selfyreports and requests for observation. Much has been 
gained be>ond the described data from extensive participant involvement in 
this evaluation study. Certainly, there is greater clarity about the goals of 
the program and the Consortium. Teachers who participated in self reports 
and observations seem to increase their awareness of their classroom 
behavior. The staff. Board members, school administrator, and teachers 
have more information about the levels of use of various components of 
validated programs, areas of greatest strength and weakness in program use, 
and the need for continuing staff development activities. Perhaps, most im 
portantly, participants seem to feel more, competent in thinking, talking, 
and asking about evaluation. It is no longer a mysterious field entered by 
outside experts, but an area in which everyone can explore, contribute, and 
understand. 



i Name. 



R 
D 



Podtion or Title Crade/Subject School and District 

Ij your teaching departracntalizcd? Ya No 

Please fill In the total number of students with whom you work using ECRI 

a. NumW of educationally handicapped or ^special pducation**' students 

b. Number of bilingual students. 

Please check the ooa meet advanced ECRI training you have attended. 

awareness intermediate 

initial advanced 

initial n other 

Please check the number of yefn you've been using ECRI. 



. this is my first year 
. this is my second year 
. thft is my third year 



. this is my fourth year 
. this is my fifth year 



A. Please check the average amount 
of time per day you use ECRI. 

O-l hours 

1-2 hours 

■ r 2-3 hours 

3-4 hours 



B. Regarding the 1:2:1 time ratio, 1 



. use it daily 
. use it weekly 



Please describes 



. use another ratio: 



7 What basal reading series or reading progranu do you use? 
8. Please check any of the follow-up statements which apply to you: 
- I use purchased mastery tests 

I use locally developed masttry tests 

I do not use master tests' 



erSq 



Stnito(S« 

0 PleMe cbedc iixlkating your use of tbe fbUowlof ttrate|iei ^ 



! 



11. 
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a. Using an infbmul reading inventory to place students in ap- 
propriate group - mayb« done by claKroom tea^ter or reading 
specialist. (Commeot) - * 

b. Testing students for mastery skills using the mastery tests and unit 
tests. (Comments) * 

c. Daily, teaching language skilll in the prescribed time ratio of 

1«0.1 /■Villa. Al^^ 1 1 .1JII«\ 

i:xti (sJaiis: practice time: bacKup sioUs) 

d. Weekly, meeting to confer individuSily with students to diagnose 
and prescribe student responses. (Comment) 

e. Daily, holding a small group discusdon to monitor student com- 
prehension and develop students* oral language skills. (Comments) 

10. Please check the dally frequency with which you teach eadi below: 










































a. new sounds 

b. new\^ords 

c. creative writing 

d. comprehension 

e. penmandUp 

f. dictation 

g. spelling 

h. study skUll • 



































































(comment) 

Check each of (he following prescribed directives which you use. « 
a. Word & Word Reoogirftion 
New Sounds 

1. sound 

New Words 

2. phonic 

3 sight 

4. word structure 1 

5. word structure 2 

6. word structure 3 

7. word structure 4 

8. word structure 5 

9. - instant error correction (words) 



b. Penmanship 

1. . penmanship - letters 

2. penmanship • words 

3. penmanship - connecting strokes 



c. Study Skills 

1 locational study skills 

2 alphabetizing 

3. surveying a book/chapt. 

4. graphic aids 

5. __ seleot topic 

6. select main idea 

7. evaluate relevance 

8. organize info, into outline 

9. following directions 

10. lUmming 

11. scanning 

12'. other (please specify)* 
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d. Dictmtioa 

1. fli}|!a untim«d 

2. fflulUi^f untlxned 

3. riff^ iy timed 

4. __ multiple timed 

ConpribfCMioo 

1. litertl oomprebendOD 

2. Infcreooes 

3. loftant error correction 

comprebeosioD 

•Ciiticil* 

4. ^ locuracy of Infa. - 

penonal experieoce 

5. accuracy of Info. - 

dieckins other material 

6. opinion vs. fact 

7. autbor's qualifications 

8 identifying fictioa- 

, famous people/evenb 

9 Idbntifyinj fiction - 

realistic fiction 

10 tdcntifyin J fiction - 

fantasy 

U. propaganda t6chniquet 

•Creative* 

12. put younelf in the story 

13. extend the story 

14. other (please specify) 
f. Sptllini 

1. part 1, trace, spell & read 

2. part 2, worA)rmation 

3. part 3, word discrimination 

(. CrMtive Writing 

1. 3 or 5 part letter ^ 

2. sentence transformation 

3. , Imagery 

4. plot development 

5. story from a picture 

(Comment) 



MANAGEMENT 

12. Checkthe^equency with which you: 

a. Respond to positive student behavior. 

b. Raipond to negative student behavior. 

c. What concerns do you have about! 

• responding to positive student behavior 

• responding to negative student behavior 



13. Check jtbe frequency with which you use etch of the foUowini me(hods for 
' setting the pace of learning in ECRI. 

a< Pitce determined by each student. ^ 

b. Pace determined by teacher. * 

c. Pace determined by each r«adlnf group. 

d. Other (please specify). 



il 



14. 



15. 



Check the number of reading groupi based on students' reading level which 
you teach in your class. 



. 1 
.2 
.3 

. 4 or more 



a. . 

b. . 

c. . 

d. . 

Describe the criteria by which you move students among these groups. 

a. 

b. 

C. ' I 



Racording 

16. Check the individual record forms which you maintain. 

a. ECRI Record Form (results of mastery tests and stu' 

competency) 

. ECRI Discussion Form 

_ ECRI Oral Reading to Teacher Form 

- Other (please specify) 

(comment) 

Check those forms which students maintain. 

ECRI Enrichment Reading 

ECRI PupU Record 

ECRI Practice Time Checklist 

ECRI Terting Card (date ready, level, date tested) 

ECRI Time ReiKilng 

Other (please spedfy) 

(comment) 



17 



idtfrfP*^ 



b. 
c. 
d. 



a. 

b. 
c. 
d. 
e. 
f. 



18. Please rate your satisfaction with E^RI. 

a. for yourself as a teacher 

b. for your students 

c. comments: 



Low 
I 


2 


3 


4 


High 
5 























ERIC 
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^ — - M»« *Ka hWn vtHi RHRI frant! 
sMMO niv9 UiV tmp jrlAI wiui bv^cu iiwuii 


Low 
1 


2 


3 


4 


High 
5 


ft. Other teftdieii 












b. prindptl 












c. other admlnistrftton (reading, supervisor* etc.) 












d. oonfortium staff 












e. students 













f. comments: 



V 20. In whftt ftrcfts do you feel the grettest need for f6Uow*up bdp ftt this time? 

r 



ERlCs 
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